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Abstract 

We study quark mass matrices in the Randall-Sundrum (RS) model with bulk symmetry 
SU(2)l x SU(2)r x U(V)b-l- The Yukawa couplings are assumed to be within an order of magni- 
tude of each other, and perturbative. We find that quark mass matrices of the symmetrical form 
proposed by Koide et. al. [Y. Koide, H. Nishiura, K. Matsuda, T. Kikuchi and T. Fukuyama, Phys. 
Rev. D 66, 093006 (2002)] can be accommodated in the RS framework with the assumption of 
hierarchyless Yukawa couplings, but not the hermitian Fritzsch-type mass matrices. General asym- 
metrical mass matrices are also found which fit well simultaneously with the quark masses and the 
Cabibbo-Kobayashi-Maskawa matrix. Both left-handed (LH) and right-handed (RH) quark rota- 
tion matrices are obtained that allow analysis of flavour changing decay of both LH and RH top 
quarks. At a warped down scale of 1.65 TeV, the total branching ratio of t — > Z + jets can be as 
high as ~ 5 x 10~ 6 for symmetrical mass matrices and ~ 2 x 10 -5 for asymmetrical ones. This 
level of signal is within reach of the LHC. 
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I. INTRODUCTION 



The idea of extra dimensions is by now a well-known one. It has led to new solutions to the 
gauge hierarchy problem without imposing supersymmetry l|, |2] , and it has opened up new 
avenues to attack the flavour puzzle in the Standard Model (SM). One such application is 
the seminal proposal of split fermions by Arkani-Hamed and Schmaltz 3| that fermion mass 
hierarchy can be generated from the wave function overlap of fermions located differently 
in the extra dimension. The split fermion scenario had been implemented in both flat extra 



dimension models 



, |4|, and warped extra dimension Randall- Sundrum (RS) models 



Subsequently, phenomenologically successful mass matrices were found in the case of one 



flat extra dimension without much fine tuning of the Yukawa couplings 



7], and in the case 



of warped extra dimensions, realistic fermion masses and mixing pattern can be reproduced 



with almost universal bulk Yukawa couplings [8l. l9l. Il0|. 

To date, many attempts in understanding the fermion flavour structure come in terms of 
symmetries. Fermion mass matrix ansatz with a high degree of symmetry were constructed 
to fit simultaneously the observed mass hierarchy and flavour mixing patterns. It is an inter- 
esting question whether in the pure geometrical setting of the RS framework where there are 
no flavour symmetries a priori, such symmetrical forms can arise and arise naturally without 
fine tuning of the Yukawa couplings, i.e. whether symmetries in the fermion mass matrices 
can be compatible with a natural, hierarchyless Yukawa structure in the RS framework, and 
to what degree. 

Another interesting and related question is whether or not one can experimentally discern 
if the fermion mass matrices are symmetric in the RS framework. In the SM, only the 
left-handed (LH) fermion mixings such as the Cabibbo-Kobayashi-Maskawa (CKM) mixing 
matrix is measurable, but not the right-handed (RH) ones. However, in the RS framework 
the RH fermion mixings become measurable through the effective couplings of the gauge 
bosons to the fermions induced from the Kaluza-Klein (KK) interactions. If the fermion 
mass matrices are symmetric, the LH and RH mixing matrices would be the same. Thus 
the most direct way of searching for the effects of these RH mixings would be through the 
induced RH fermion couplings in flavour changing processes that are either not present or 
very much suppressed in the SM. 

In this work we study how well the RS setting serves as a framework for flavour physics 
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either with or without symmetries in the fermion mass matrices, and if the two scenarios 
can be distinguished experimentally. We concentrate on the quark (and especially the top) 
sector, and we study the issues involved in the RSI model Q with an SU(2) L x SU(2) R x 
U(l)x bulk symmetry, which we shall refer to as the minimal custodial RS (MCRS) model. 
The U(l)x is customarily identified with U(1)b-l- The enlarged electroweak symmetry 
contains a custodial isospin symmetry which protects the SM p parameter from receiving 
excessive corrections, and the model has been shown to be a complete one that can pass all 
electroweak precision tests (EWPT) at a scale of ~ 3 to 4 TeV [11]. 

The organization of the paper is as follows. In Sec. [IT] we quickly review the details of 
the MCRS model to fix our notations. In Sec. II III we investigate which type of mass matrix 
ansatz is compatible with Yukawa couplings that are perturbative and not fine-tuned by 
matching the ansatz form to that in the MCRS model. Relevant matching formulae and 
EWPT limits on the controlling parameters are collected into the two Appendices. We 
also investigate possible patterns in the mass matrices by numerically scanning the EWPT 
allowed parameter space for those that can reproduce simultaneously the observed quark 
masses and the CKM mixing matrix. In Sec. HVI we study the effects of quark mass matrices 
being symmetrical or not are having on flavour changing top decays, t — > c(u)Z, which are 
expected to have the clearest signal at the LHC. We summarized our findings in Sec. [V] 



II. REVIEW OF THE MCRS MODEL 



In this section, we briefly review the set-up of the MCRS model. We summarize relevant 
results on the KK reduction and the interactions of the bulk gauge fields and fermions, and 
establish the notation to be used below. 



A. General set-up and gauge symmetry breaking 

The MCRS model is formulated in a 5-dimensional (5D) background geometry based 
on a slice of AdS$ space of size nr c , where r c denotes the radius of the compactified fifth 
dimension. Two 3-branes are located at the boundaries of the AdS§ slice, which are also the 
orbifold fixed points. They are taken to be = (UV) and <p = n (IR) respectively. The 
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metric is given by 

ds 2 = G AB dx A dx B = e" 2 ^ ri^dx»dx u - r 2 d<j) 2 , a(<f>) = kr c \<f)\ , (1) 

where rj^ = diag(l, —1, —1, —1), k is the AdS 5 curvature, and — n < < ir. 

The model has SU (2) l x SU {2)r x U{1)x as its bulk gauge symmetry group. The fermions 
reside in the bulk, while the SM Higgs, which is now a bidoublet, is localized on the IR brane 
to avoid fine tuning. The 5D action of the model is given by 

S = J d A xJ^ d(j)VG[£ g + C f + C uv 5(0) + Cm 5{<j) - tt)] , (2) 

where L g and C / are the bulk Lagrangian for the gauge fields and fermions respectively, and 
Cjr contains both the Yukawa and Higgs interactions. 
The gauge field Lagrangian is given by 

C g = -~ <W AB W AB + W AB W AB + B AB B AB ) , (3) 

where W, W, B are field strength tensors of SU(2) L , SU{2) R and U{l)x respectively. On 
the IR brane, SU{2) L x SU{2) R is spontaneously broken down to SU{2) V when the SM 
Higgs acquires a vacuum expectation value (VEV). On the UV brane, first the custodial 
SU{2)r is broken down to U{1)r by orbifold boundary conditions; this involves assigning 
orbifold parities under Si/(Z 2 x Z' 2 ) to the /i-components of the gauge fields: one assigns 
( — h) for W^ 2 , and (++) for all other gauge fields, where the first (second) entry refers to 
the parity on the UV (IR) boundary. Then, U(1)r x U(1)x is further broken down to U(l)y 
spontaneously (via a VEV), leaving just SU{2) L x U(l)y as the unbroken symmetry group. 

B. Bulk gauge fields 

Let A M (x, 0) be a massless 5D bulk gauge field, M — 0, 1, 2, 3, 5. Working in the unitary 
gauge where A 5 = 0, the KK decomposition of A^(x, 0) is given by (see e.g. 

A t (^0) = ^E4 n) O*)Xn(0), ( 4 ) 
v n 

where Xn are functions of the general form 

Xn = W l Jl ( Znea ) + hijn n )Y 1 {z n e a )\ , z n = , (5) 



that solve the eigenvalue equation 

—8$ e~ 2a d4, -ml) X n = 0, (6) 

' c / 

subject to the orthonormality condition 

r 

d<j> XnXm = S m n • (7) 



Depending on the boundary condition imposed on the gauge field, the coefficient function 
&i( m n) 1S given by 

(++)B.C 6l(m „) = - T ^ I _^ y = -_, (8) 

(- + )B.C. Mmn) = - w -^ )= -—, (9, 

which in turn determine the gauge KK eigenmasses, m n . For fields with the (++) boundary 
condition, the lowest mode is a massless state with a flat profile 

Xo = l, (10) 

while no zero-mode exists if it is the ( — h) boundary condition. The SM gauge boson is 
identified with the zero-mode of the appropriate bulk gauge field after KK reduction. 

C. Bulk fermions 

The free 5D bulk fermion action can be written as (see e.g. 0, Q]) 

S f = Jd'x^dcpVG^ 1 ^ 7 a (^-S)^] +msgn(0)Mj , (11) 

where 7 a = (7 M ,i7 5 ) are the 5D Dirac gamma matrices in flat space, G is the metric given 
in Eq. (HD, the inverse vielbein, and m = c k is the bulk Dirac mass parameter. There is 
no contribution from the spin connection because the metric is diagonal [5:1. The form of the 

n 

mass term is dictated by the requirement of Z2 orbifold symmetry [5]. The KK expansion 
of the fermion field takes the form 

3<t/2 00 

*l,rM) = ^E^C*)/^). (12) 
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where the subscripts L and R label the chirality of the fields, and f£ R form two distinct 
sets of complete orthonormal functions, which are found to satisfy the equations 



-d. 



<t> 



+ c)k 



f n R = m n e°fl 



r c \2 

with the orthonormality condition given by 



r L = ^r R) (13) 



(14) 



Of particular interest are the zero-modes which are to be identified as SM fermions: 

//U(0> c l,r) = 



MlTM e (l/2 T e 1 , fl )fc^ 
e fcr c 7r(l=F2c LjH ) _ ^ 



(15) 



where the upper (lower) sign applies to the LH (RH) label. Depending on the Z 2 parity of 
the fermion, one of the chiralities is projected out. It can be seen that the LH zero mode is 
localized towards the the UV (IR) brane if cl > 1/2 (cl < 1/2), while the RH zero mode is 
localized towards the the UV (IR) brane when c R < —1/2 (cr > —1/2). 
The higher fermion KK modes have the general form 



pn 
JL. 



R 



-B a (z n e a ) , B a (z n e a ) = J a (z n e a ) + b a (m n )Y a (z n e a ) , 



(16) 



where a = |c± 1/2| with the LH (RH) mode takes the upper (lower) sign. Depending on 
the type of the boundary condition a fermion field has, the coefficient function b a (m n ) takes 



the form 



Hi 



b a (m n ) 
b a (m n ) 



(++) B.C. 

(-+) B.C. 

and normalization factor can be written as 

e M4>) 



<T(7r) x 



<t(tt) s 



Y aTl {z n e^) 



:++) B.C. : 
-+) B.C. : 



Nl 



2kr c 7i 
1 

2kr r 7T 



B 2 Jz n e 



(f>=TT 

0=0 



[e^Bl{z n e^)-Bl Tl {z n )] 



(17) 
(18) 

(19) 
(20) 



The upper sign in the order of the Bessel functions above applies to the LH (RH) mode when 
Cl > —1/2 (cr < 1/2), while the lower sign applies to the LH (RH) mode when cl < —1/2 
(cr > 1/2). The spectrum of fermion KK masses is found from the coefficient function 
relations given by Eqs. ffTTl) and ffTB"]) . 



Now there is an additional SU(2)r gauge symmetry over the SM in the bulk, and the 
fermions have to be embedded into its representations. Below we chose the simplest way of 
doing this, viz. the LH fermions are embedded as SU(2)r singlets, while the RH fermions 



are doublets 



Note that since the SU(2)r is broken on the UV brane by the orbifold 
boundary condition, one component of the doublet under it must be even under the Z 2 parity, 
and the other odd. This forces a doubling of RH doublets where the upper component, say 
the up-type quark, of one doublet, and the lower component of the other doublet, the down- 
type, are even. 

D. Fermion interactions 

In 5D, the interaction between fermions and a bulk gauge boson is given by 

SffA = 9$ j d 4 x d<f> VGE^ a A M y + h. c. , (21) 

where g$ is the 5D gauge coupling constant. After KK reduction, couplings of the KK 
modes in the 4D effective theory arise from the overlap of the wave functions in the bulk. In 
particular, the coupling of the mth and nth fermion KK modes to the gth gauge KK mode 
is given by 

/*7T 

mnq 9& I ji rm rn 9§ /oo\ 

9 f fA = — / d <f> fL,RfL,RXq , 9i = ~y= , (22) 



'o V' c/ 
where g 4 = g$M is the 4D SM gauge coupling constant. Note that since the gauge zero-mode 

has a flat profile (Eq. ([TO]) ), by the orthonormality condition of the fermions wave functions, 

Eq. ([H]), only fermions of the same KK level couple to the gauge zero-mode, and the 4D 



coupling is simply given by gj'j™ = 5'4- 

With the Higgs field $ localized on the IR brane, the Yukawa interactions are contained 
entirely in Cm of the 5D action ([2]). The relevant action on the IR brane is given by 

S Yuk = [d A xd(f)VG5((f)-TT)^^ i (x,(l))^ 1 (x,(P)^(x)+h.c., (23) 
J kr c 

where X^^j are the dimensionless 5D Yukawa coupling, and i, j the family indices. Rescaling 
the Higgs field to H(x) = e~ kTc7T $(x) so that it is canonically normalized, the effective 4D 
Yukawa interaction obtained after spontaneous symmetry breaking is given by 

S'Yuk = J d 4 xv w ^ ]T ^\x)^f R \x)fn^ cft/gfa cf) + h. c. , (24) 



m,n 



where (H) = vw = 174 GeV is the VEV acquired by the Higgs field. The zero modes give 
rise to the SM mass terms, and the resulting mass matrix reads 

(Mf%- = vw^f^c^f^c*) = vw^F L (c L f jF R (cl) , f = u, d , (25) 

where the label / denotes up-type or down-type quark species. Note that the Yukawa 
couplings are in general complex, and so take the form = p^e 1 ^ , with p{j, <pij the 
magnitude and the phase respectively. 



III. STRUCTURE OF THE QUARK MASS MATRICES 

In this section, we investigate the possible quark flavour structure in the RS framework. 
One immediate requirement on the candidate structures is that the experimentally observed 
quark mass spectrum and mixing pattern are reproduced. Another would be that the 5D 
Yukawa couplings are all of the same order, in accordance with the philosophy of the RS 
framework that there is no intrinsic hierarchy. We also required that constraints from EWPT 
are satisfied. 

To arrive at the candidate structures, we follow two strategies. One is to start with a 
known SM quark mass matrix ansatz which reproduces the observed quark mass spectrum 
and mixing pattern. The ansatz form is then matched onto the RS mass matrix to see if 
the above requirements are satisfied. The other strategy is to generate RS mass matrices at 
random and then pick out those that satisfy the requirements above. 1 

To solve the hierarchy problem, we take kr c = 11.7 and the warped down scale to be 
k = /ce~ fcrc7r = 1.65 TeV. Since new physics first arise at the TeV scale in the RS framework, 
it is also where experimental data are matched to the RS model below. We will assume that 
the CKM matrix evolves slowly between \x = Mz and \i — 1 TeV so that the PDG values 
can be adopted, and we will use the running quark mass central values at \i = 1 TeV from 
Ref. [l3|. 

~ n, 

This has been tried before in Ref. [8J, but it was done for the case with itikk > 10 TeV where there is a 
little hierarchy. 



s 



A. Structure from mass matrix ansatz 



In trying to understand the pattern of quark flavour mixing, many ansatz for the SM 
quark mass matrices have been proposed over the years. There are two common types of 
mass matrix ansatz consistent with the current CKM data. One type is the Hermitian ansatz 

accommodate \VJ 113- The other type is the symmetric ansatz proposed by Koide * 
al. [16|, which was inspired by the nearly bimaximal mixing pattern in the lepton sector. 2 
Using these ansatz as templates, we find that only the Koide-type ansatz admit hierarchy- 
free 5D Yukawa couplings; this property is demonstrated below. That Fritzsch-type ansatz 
generically lead to hierarchical Yukawa couplings is shown in Appendix |A] 
The admissible ansatz we found takes the form 



M 



f 



P}M f Pj 



f = u, d, 



(26) 



where Pf = diag{e i5 i, e lS z , e l5 s} is a diagonal pure phase matrix, and 



M 



f 



(27) 



\ c f B f A f J 



c f A f B f 
If 

with all entries real and much less than all other entries. When £j = 0, the ansatz of 
Ref. (l6| is recovered. 

The real symmetric matrix Mf is diagonalized by the orthogonal matrix 



0]MfOj 



f\{ 0^ 

xt 



\{j 



( 



Oi 







s f 



\ 



_£L L £L 

s f l c f 

\ V2 y/2 V2 



where the eigenvalues are given by 



1 2 



A f — B 



B f + z f - Jsc) + (A f + B f - t,y 



f ' 



A 4 



Bf + if + JSC* + (A f + B f - z f y 



(28) 



(29) 



2 In the SM, because of the freedom in choosing the RH flavour rotation, quark mass matrices can always 
be made Hermitian. But this need not be the case in the RS framework as we show below. 
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and the mixing angles are given by 



a{ - a{ y a{ - a{ 

Note that the components of Mf can be expressed as 

Bf = l(\ f 3 + \ f 2 +\{-df), 



(30) 



(31) 



To reproduce the observed mass spectrum m{ < m } 2 < 777.3, the eigenvalues \\, i — 1, 2, 3, 
are assigned to be the appropriate quark masses. For the Koide ansatz (the £/ = case), 
it was pointed out in Ref. [l7|] that different assignments are needed for the up and down 
sectors to fit \V u b\ better. Since the ansatz, Eq. ( l26l) . is really a perturbed Koide ansatz, we 
follow the same assignments here: 



A? 



-777 



1 i 



A2 



777 



2 • 



A 3 



777 



3 ! 



m 



3 • 



m 



(32) 



Now since OjMdOd = diag{— mf, mf, m 2 } for the down-type quarks, to put the eigen- 
values into hierarchical order, the diagonalization matrix becomes 0' d = OdT 23 , where 

/ 1 o\ 



23 



1 
yd 10, 



(33) 



The quark mixing matrix is then given by 

/ 

V ■ — (1 T P P^O' — 



-as d ac d k 



(34) 



where 



k = -(e 103 + e 



a 



(e i5s - e i&2 \ 



6i = %-6t, 7 = 1,2,3. 



(35) 



Without loss of generality, Si is taken to be zero. 
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The matrix V m i x depends on four free parameters, #2,3 and A good fit to the CKM 
matrix is found by demanding the following set of conditions: 

\ K \ = \V cb \ = 0.04160 , \a\s u = \V ub \ = 0.00401 , \a\s d = \V cd \ = 0.22725 , (36) 
and 5c p = — (63 + 82)/ 2 = 59°. These imply 

S 2 = -2.5 5 8 93 , 5 3 = -0.49 9 44 , £ u = 1.36226 x 10" 3 , & = 6.50570 x 10" 5 , (37) 
which in turn lead to a Jarlskog invariant of J = 3.16415 x 10~ 5 and 

( 0.97380 0.22736 0.00401^ 



0.22725 0.97294 0.04160 
yO.00816 0.04099 0.99913 j 



(38) 



both of which are in very good agreement with the globally fitted data. 
With 5 Uj d determined, so are M U)d also. From Eq. f[3"Tl) we have 

A u = 77.32226 , B u = 76.77526 C u = 0.43733 , 

A d = 1.26269 , B d = -1.21731 C d = 7.91684 x 10" 3 . (39) 



Parameters of the RS mass matrix (J25J) can now be solved for by matching the RS mass 
matrix onto the ansatz (|26|) . Starting with M^ s , there are a total of 24 parameters to be 
determined: six fermion wave function values, F^(cqJ and ^(c^J, nine Yukawa magni- 
tudes, p^-, and nine Yukawa phases, 0^, where i, j = 1, 2, 3. 3 Matching M^ s to M u results 
in nine conditions for both magnitudes and phases. Thus all the up-type Yukawa phases 
are determined by the three phases <5", while six magnitudes are left as free independent 
parameters. These we chose to be Fl(cq 3 ) and Fr(cjj 3 ), which are constrained by EWPT, 
and p\ x , p\\, p\, p\ 2 . 

Next we match Mf s to M d . Since F^cqJ have already been determined, there are only 
21 parameters left in M^ 5 : Fr(cdJ, pfj, and (pfj. Again all the down-type Yukawa phases are 
determined by the three phases, 5f, leaving three free magnitudes which we chose to be pf x , 
P32, and P33. We collect all relevant results from the matching processes into Appendix iBl 



3 We will denote using subscripts Q, U, and D respectively, for the left-handed quark doublet, and the 
right-handed up- and down-type singlets of S'[/(2)l. 
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To see that the ansatz (126)) does not lead to a hierarchy in the Yukawa couplings, note 
from Eq. fl3T|) we have 



A f ~\B f \~?f, C u ~^f^\ C d ~^f^. (40) 
Given this and Eq. (157)) . we see from Eqs. (IBID and (lB4h that as long as 

031 ~ P32 ~ P33 ~ Pll ~ P21 ~ P31 ~ P32 ~ P33 > l 4i ) 

all Yukawa couplings would be of the same order in magnitude. It is amusing to note that if 
we begin by imposing the condition that the 5D Yukawa couplings are hierarchy-free instead 
of first fitting the CKM data, we find 



«~10- 3 , U~ a/^2^h/^~3x 10- 5 , (42) 



m 3 



which give the correct order of magnitude for £ U) <2 necessary for V m ^ to fit the experimental 
CKM values. 

From relations (IB1[) . (IB3I) . and f)B6|) . for mass matrices given by the ansatz (126)) . all local- 
ization parameters can be determined from just that of the third generation SU{2) L doublet, 
cq 3 , and the Yukawa coupling magnitudes listed in Eq. (14T1) . To satisfy the bounds from 
flavour- changing neutral-currents (FCNCs), LH light quarks from the first two generations 
should be localized towards the UV brane. As discussed in Appendix [Bj for generic choices 
of Yukawa couplings this is so for the first generation LH quarks, but not for the second 
generation. In order to have Cq 2 > 0.5 while still satisfying constraints from Eqs. (1B8P 
and (IB9P and the EWPT constraint Cjj 3 < 0.2, we choose 



^ = 0.2615, ^=^ = 0.7, p^3 = 0-85, ^, = ^=^=^ = 1. (43) 
P21 

We also have to shorten the EWPT allowed range of cq 3 to (0.3,0.4) so that cq 2 > 0.5 
is always satisfied. Note that relation ()B3j) constrains cjj 2 to be greater than —0.5 if the 
perturbativity constraint, A5 < 4, is to be met. 

The localization parameters increase monotonically as cq 3 increases. Except for cy 2 3 , the 
variation of the localization parameters is small. We list below their range variation as cq 3 
varies from 0.3 to 0.4 given the choice of the Yukawa couplings (|43)) : 

0.65 < c Ql < 0.66, 0.50 < c Q , 2 < 0.52, 
-0.62 < c Vl < -0.61 , -0.26 < c U2 < -0.01 , -0.16 < c Ua < 0.18 , 

-0.75 < c Dl < -0.74 , -0.60 < c D2 Z < -0.59 . (44) 
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B. Structure from numerical search 



The RS mass matrix given by Eq. (|25|) has a productlike form: 



^ aib\ aib 2 ' 



M 



RS 



Fr(c 



a 2 b\ a 2 b 2 a 2 b 3 
a 3 b 2 a 3 b 3 J 

and it can be brought into the diagonal form by a unitary transformation 



(45) 



{UtfMf 8 U f R 



(\{ 0^ 
A{ 



A{ 



f = u, d. 



(46) 



Suppose there is just one universal 5D Yukawa coupling, say A5 = 1, then the RS mass matrix 
Mj? s would be singular with two zero eigenvalues, and both the LH and RH quark mixing 
matrices would be the identity matrix, i.e. = (U^ R = 1 3X 3- Thus, in order to 

obtain realistic quark masses and CKM mixing angles (V£ ix = Vckm), one cannot assume 
one universal Yukawa coupling. Rather, for each configuration of localization parameters, the 
magnitudes and phases of the 5D Yukawa coupling constants, pij and 4>iji will be randomly 
chosen from the intervals [1.0, 3.0] and [0, 2tt] respectively, and we take a sample size of 10 5 . 

The numerical search is done with 0.5 < cq 12 < 1 and — 1 < cjj 12 , cd 123 < —0.5 so 
that the first two generation quarks, as well as the third generation RH quarks of the D 3 
doublet are localized towards the UV brane. For the third generation, 0.25 < cq 3 < 0.4 
and —0.5 < qy 3 < 0.2 are required so the EWPT constraints are satisfied (see Appendix IB"]) . 
We averaged the quark masses and CKM mixing angles over the entire sample for each 
configuration of localization parameters, and these choices yielded averaged values that are 
within one statistical deviation of the experimental values at \i — 1 TeV as given in Ref. 13]. 
Below we give three representative configurations from the the admissible configurations 
found after an extensive search. 



Configuration I: 



c Q = {0.634,0.556,0.256}, 
CJJ = {-0.664, -0.536,0.185}, 
c D = {-0.641, -0.572, -0.616} . 



(47) 
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In units of GeV, the mass matrices averaged over the whole sample are given by 



i.97 x KT 4 0.049 0.767 
0.010 0.554 8.69 
9.06 142.19 



(\M d \) 



J 



^ 07 v 1 n-4 n n/iQ n 7R7 \ 

(\M U \) = 

K 0.166 
which have eigenvalues 

m t = 109(52) , m c = 0.56(59) , 

m b = 2.59 ± 1.11, m s = 0.048(32) , 

The resulting mixing matrices are given by 



0.0019 0.017 0.0044 
0.022 0.196 0.050 
0.352 3.209 0.813 



\V U L S \ = 0.16(14) 



V U R S \ = 0.42(24) , 



ub I 

rR 



0.009(11) , 



V^|=0.12(10), 



m u = 0.0011(12) , 
m d = 0.0017(12) . 

\V£\= 0.079(74) , 
1^1=0.89(13), 



which give rise to an averaged Jarlskog invariant consistent with zero with a standard 
of 1.3 x 10~ 4 . 

• Configuration II: 

c Q = {0.629,0.546,0.285}, 
dj = {-0.662, -0.550, 0.080} , 
c D = {-0.580, -0.629, -0.627} . 

In units of GeV, the mass matrices averaged over the entire sample are given by 



( 0.0011 0.039 0.834^ 



\M U \) 



\ 



1 0.017 0.0034 0.0036^ 



0.014 0.492 10.55 

0.16 5.726 122.87 

which have eigenvalues 

m t = 95(45) , m c = 0.49(50) , 

m b = 2.01(83) , m s = 0.057(35) , 

The resulting mixing matrices are given by 

\V U L S \= 0.14(12), lK.il = 0.011(13), 

1^= 0.30(20), |V^|=0.90(12), 



0.209 0.043 0.046 
2.43 0.506 0.539 



m u = 0.0014(16) , 
m d = 0.0022(15). 



|V^| = 0.11(10) , 
|Kf|= 0.23(15), 
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which give rise to an averaged Jarlskog invariant consistent with zero with a standard error 
of 2.3 x 10" 4 . 



Configuration III: 



c Q = {0.627,0.571,0.272}, 
Cu = {-0.518,-0.664,0.180}, 
c D = {-0.576, -0.610, -0.638} , 



(55) 



In units of GeV, the mass matrices averaged over the entire sample are given by 



( 0.092 0.0010 0.940^ 



(\M U \) 



0.554 0.0065 5.66 
13.4 0.158 136.9 



f 0.019 0.0066 0.0026^ 



(\M d \) 



0.114 0.039 0.016 
2.774 0.955 0.376 



which have eigenvalues 

m t = 106(50) , 
m b = 2.32(94) , 



m c = 0.56(55) , 
m s = 0.036(21) , 



m u = 0.0013(12), 
m d = 0.0023(16). 



The resulting mixing matrices are given by 



|^|=0.27(19), 



V U R S \ = 0.77(19) , 



IKil = 0.010(10), 

|\/^l=0.36(21), 



\V£\= 0.048(44) , 



V£\ = 0.85(15) , 



(56) 



(57) 



(58) 



which give rise to an averaged Jarlskog invariant consistent with zero with a standard error 
of 1.9 x 10~ 4 . 

In summary, from the numerical study we found that in the RS framework, there is 
neither a preferred form for the mass matrix nor a universal RH mixing pattern. Note that 
the RH mixing matrix is in general quite different from its LH counterpart, viz. the CKM 
matrix. 

IV. FLAVOUR VIOLATING TOP QUARK DECAYS 

In this section we study the consequences the different forms of quark mass matrices have 
on FCNC processes. We focus below on the decay, t — > c(u) Z — > c(u) 11, where I — e, ji, r, v. 
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Modes which decay into a real Z and c (w)-jets are expected to have a much higher rate than 
those involving a photon or a light Higgs, which happen through loop effects. Moreover, 
much cleaner signatures at the LHC can be provided by leptonic Z-decays. 



A. Tree-level flavour violations in MCRS 

Tree-level FCNCs are generic in extra-dimensional models, for both a flat background 
geometry [18| and a warped one . Because of the KK interactions, the couplings 

of the Z to the fermions are shifted from their SM values. These shifts are not universal in 
general, and so flavour violations necessarily result when the fermions are rotated from the 
weak to the mass eigenbasis. 

More concretely, consider the Zff coupling in the weak eigenbasis: 

£nc D 9zZ, i Qz(h) ^(Sij + K^fiL-ffjL + Qz(fn) zZiSij + 4)fi*ffiR \ , (59) 
where i, j are family indices, re^ = diag(/«i, k 2 , k 3 ), and 

QzU) = Tl(f) - s 2 Q f , Q f = Tl(f) + T*(f) + Q x (f) = !*(/) + ^ , (60) 

with Qf is the electric charge of the fermion, Yf/2 the hypercharge, T^R^f) the weak 
isospin under SU(2)l,r, and Qx(f) the charge under U(l)x- We define «y = 5gfj R /gz to 
be the shift in the weak eigenbasis Z couplings to fermions relative to its SM value given by 
gz = e/(sc), as well as the usual quantities 

9l9' , gRgx e 



• ,~ j~t ■ 9' = ^-, — 2"' s = — , c = Vl - s 2 , (61) 

Vdl + 9 12 9 2 r + 9x 9l 

where gi = g 5 L/ ' s/r^K is the 4D gauge coupling constant of SU{2) L (and similarly for the 
rest). Rotating to the mass eigenbasis of the SM quarks defined by /' = Wf, where the 
unitary matrix U diagonalizes the SM quark mass matrix, flavour off-diagonal terms appear: 

£fcnc ^ 9zZn 

L a,b a,b J 

where the mass eigenbasis flavour off-diagonal couplings are given by 

E^U)-4'V^u- (63) 



~L,R 
K ab 



i -J 
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Note that the off-diagonal terms would vanish only if k is proportional to the identity matrix. 

In the RS framework, one leading source of corrections to the SM neutral current inter- 
action comes from the exchanges of heavy KK neutral gauge bosons as depicted in Fig. [TJ 
The effect of gauge KK exchanges give rise only to the diagonal terms of k. It can be effi- 



/ <H > 




FIG. 1: Correction to the Zff coupling due to the exchange of gauge KK modes. The fermions 
are in the weak eigenbasis, and X = Z, Z'. 



ciently calculated with the help of the massive gauge 5D mixed position-momentum 



propagators, which automatically sums up contributions from all the KK modes 
The leading contributions can be computed in terms of the overlap integral, 
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space 



2l| 



r c d<p\f L)R (<p,c LyR )\ 2 G p=0 (<p^) 



(64) 



where G p=0 is the zero-mode subtracted gauge propagator evaluated at zero 4D momentum. 
For KK modes obeying the (++) boundary condition, G p= q is given by |2l| 



Ak(kr c 7r) 



7 2kr c TT 



kr c ir 



+ e 2kr ^<(l-2kr. 



_|_ e 2kr c cj> > 



1 + 2kr c (ir 



and those obeying the ( — h) boundary condition 

1 



2k 



1) 



(65) 



(66) 



where 



>) is the minimum (maximum) of 6 and 0'. The gauge KK correction to the Z 



coupling is thus «£• = with n 9 given by 



e 2 I q" 

K)l,R = -53 { G ++ 



22] 



G 



Il.r 



(67) 



I Qz(qi, R ) 

where the label q denotes the fermion species. Note that when the fermions are localized 
towards the UV brane (c^ > 0.6 and c R < —0.6), G_ + is negligible, while G ++ becomes 



essentially flavour independent 



22 
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Another source of corrections to the Zff coupling arises from the mixings between the 

fermion zero modes and the fermion KK modes brought about by the the Yukawa interac- 
tions. These generate diagonal as well as off-diagonal terms in k. The diagram involved is 
depicted in Fig. [2J 

< H > < H > 

\ ? i 

\ } i 

\ S / 

X ) 7 ' 

\ (A I 

\ ) I 

\ ) / 

\ s / 



X > r >)( 



fi fKK fKK fj 

J a J a 



FIG. 2: Correction to the Zff coupling due to SM fermions mixing with the KK modes. The 
fermions are in the weak eigenbasis. 

The effects of the fermion mixings may be similarly calculated by using the fermion 
analogue of the gauge propagators. It is however much more convenient to deal directly 
with the KK modes here. The KK fermion corrections to the weak eigenbasis Z couplings 
can be written as 

(4>-££^*s, (4)-EE^, (68) 

a n=l K n > a n=l v n! 

where m n is the nth level KK fermion mass, mj a are entries of the weak eigenbasis RS mass 
matrix (125]) with a a generation index 4 , and 

/jU(^c^) 2 Q z (f R , L ) 



3 a 



R,L 



JR 



R,L\ 



QzUl,r) ' 



(69) 



with the argument of Qz, f = u, d, denoting up-type or down-type quark species. Note 
that for c L a < 1/2 and > -1/2, |/£ ijR (tt, c% r )\ « v / 2fc^F. 

To determine k a b in Eq. ( J63l) . one needs to know the rotation matrices Ul and Ur. In the 
case where the weak eigenbasis mass matrices are given by the symmetric ansatz ( 1261) . the 
analytical form of the rotation matrices are known. By rephasing the quark fields so that 
8f = and all the Yukawa phases reside in down sector, the up-type rotation matrix is just 



4 For shift in the LH couplings, the index a runs over the generations of both types of SU(2)r doublets, U 
and D, both of which contain KK modes that can mix with LH zero modes. For shift in the RH couplings, 
a runs over just the generations of the only type of S , J7(2)l doublets, Q. 
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the orthogonal diagonalization matrix given by Eq. (128]) . Using the solution of the CKM fit 
given in Eq. fl37|) . we have 



V 



0.99999 0.00401 
-0.00284 0.70710 
-0.00284 ^= 0.70710 



(70) 



Since we are interested in flavour violating top decays, the relevant mass eigenbasis off- 



diagonal corrections are k 3r 



k y 3r + k 



f 

3r > 



1, 2. For the discussion below, using re- 



lations ( 1B1I) . (lB3p . and ( 1B6I) we will trade the dependences of k a ^ on all the different 
localization parameters for just a single dependence on cq 3 , and the Yukawa coupling mag- 
nitudes which we fix to take the values given in Eq. (14*3]) . Recall that with this choice of the 
Yukawa coupling magnitudes, the EWPT allowed range for cq 3 is between 0.3 and 0.4. 
Since k\- = Kq t 5ij, the gauge KK contributions is simply k 3r = ^\ ^ 9 qi {U u )\ i U] l r , with 



h tu 



2.00672 x 10~ 3 (2k 9 u - k 9 c - K 9 t ) 



k 9 



0.50 {k 9 - «g) 



(71) 



We plot function of cq 3 in Fig. [3j 



RH 



0.0001- 



).32 0.34 0.36 0.38 0.4 



C Q 3 




(a.) 



(b) 



FIG. 3: Gauge KK contribution in the case of symmetrical mass matrices to (a) km and (b) kf C 
The labels LH and RH indicate whether it is for the LH or RH coupling. 



For the fermion KK contributions, since the decoupling of the higher KK modes is very 
efficient, hence just the first KK mode provides a very good approximation to the full tower. 
We plot using this approximation \k^ r \ as functions of cq 3 in Fig. HI 
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1*4 1 



\*L\ 



0.0002 



).32 0.34 0.36 0.38 



C Q 3 



.32 0.34 



.36 0.38 0.4 



C Q 3 



(a) (b) 

FIG. 4: Fermion KK contribution in the case of symmetrical mass matrices to (a) kt u and (b) kt c - 
The labels LH and RH indicate whether it is for the LH or RH coupling. The plots are made using 
the first KK mode to approximate the full KK tower. 

B. Experimental signatures at the LHC 



The branching ratio of the decay t — > c(u)Z is given by 



Br(t - c{u)Z) = ^(\Qz(t L ) fc£ (u) | 2 + \Qz(t R ) kf c{u) \ 2 



l-x t 



1 + 2x t \ y t 



(72) 



cv~ / \l-y t J \l+2y t J x t 

where x t = m 2 z /m 2 and y t = m^/m 2 . In Fig. Owe plot the branching ratio as a function 
of cq 3 in the case where the weak eigenbasis mass matrix has the symmetric ansatz form 
of ( 1261) . It is clear that the dominant channel is t — > cZ. The branching ratio is at the 

Br(i ^ uZ) Br (* ^ cZ) 




.32 0.34 . 36 0.38 0.4 



C «3 



.32 0.34 0.36 



.38 . 



C Q 3 



(a) (b) 

FIG. 5: Branching ratio in the case of symmetrical mass matrices as a function of cq 3 for the decay 
(a) t — > uZ and (b) t — > cZ. The labels LH and RH indicate LH or RH top decay. 

level of a few 10~ 6 , which is to be compared to the SM prediction of O(10~ 13 ) j^. As cq 3 
increases, the decay changes from being mostly coming from the LH tops at the low end of 
the allowed range of cq 3 , to having comparable contributions from both quark helicities at 
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the high end. Note that one can in principle differentiate whether the quark rotation is LH 
or RH by studying the polarized top decays. 

For the case of asymmetrical quark mass matrix configurations found in Sec. IIII B\ the 
resultant branching ratios and the associated gauge and fermion KK flavour off-diagonal 
contributions are tabulated in Table [B We give results only for the decay into charm quarks 
since this channel dominates over that into the up-quarks. The magnitude of our branching 
ratios for both cases of symmetrical and asymmetrical quark mass matrices are consistent 



with previous estimate in the RS framework 241 ]. 



TABLE I: Branching ratios of t — > c Z and the associated gauge and fermion KK flavour off-diagonal 
contributions for the case of asymmetrical mass matrices found from numerical searches. 



Config. 


1 ~ 9 1 




1 "9 1 
\ K R\ 




\k f \ 
\ k l\ 


14? 




Br(ii) 


Br(t fl ) 


I 


3.5 x 10" 


-4 


7.7 x 10" 


-3 


8.2 x 10~ 3 


4.7 x 10- 


-3 


1.4 x 10~ 5 


4.1 x 10~ 6 


II 


4.3 x 10" 


-4 


5.8 x 10- 


-3 


9.9 x 10~ 3 


2.9 x 10- 


-3 


2.1 x 10~ 5 


2.0 x 10~ 6 


III 


2.1 x 10" 


-4 


3.8 x 10- 


-3 


5.0 x 10~ 3 


7.0 x 10- 


-3 


5.4 x 10~ 6 


3.2 x 10~ 6 



It is interesting to note from Fig. 0(b) and Table [I] that in t — > c Z decays, the LH decays 
dominate over the RH ones in the case of both symmetrical and asymmetrical quark mass 
matrices. The reason for this is however different for the two cases. In the symmetric case, 
M u = Ml and so 17% = U R = U u . Thus the difference between the LH and RH decays 
is due to the differences in the weak eigenbasis couplings, as can be seen from Eq. (171I) . 
and Qz- By comparing Fig. IU(b) to EJ^b) we see |re tc | ~ l^fj, and from Fig. [3](b) we have 
0-9 < \{k 9 tc ) R \/\(k 9 tc ) L \ < 2 5 . However, as \Q z (t L )\ > \2Q z (t R )\, the net effect is that the 
LH decay dominates (see Eq. (f72|) ). 

In the asymmetrical case, M u ^ Ml and Uf, ^ U R with no pattern relating the LH to the 

5 It may seem counterintuitive that |(k? c ).r| can be smaller than |(/tf c )i| (for cq 3 < 0.32), as one may expect 
that the couplings to be dominated by the top contribution, and the coupling to the RH top to be larger 
than that to the LH top due to the fact that the RH top is localized closer to the IR brane. However, 
such expectations can be misleading. Because of the mixing matrices, the mass eigenbasis coupling, /tf c , 
is not just a simple sum of the weak eigenbasis couplings, n 9 qi , but involves their differences as already 
mentioned. Moreover, although the greatest contribution comes from the top, the contribution from 
the second generation may not be completely negligible, as is the case here for (k£)r for the particular 
symmetric ansatz that we study. 
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RH mixings. In each of the configurations of localization parameters listed in Table [H while 

K«?cUI < K^tJfll) ^ turns out that not onl y l(«f c )fl| ~ \(kL)r\ and ~ l(«fc)fl|) there 

is also a relative minus sign between the gauge and the fermion KK contributions, which 
results in a destructive interference that leads to a greater branching ratio for the LH decay. 
This is to be contrasted with Ref. [24| where it is the RH mode that is found to dominate. 
There it appears that the possibility of having a cancellation between the gauge and fermion 
KK contributions was not considered. 

We note and emphasize here the crucial role the quark mass and mixing matrices play in 
determining the mass eigenbasis flavour off-diagonal couplings k a ^. Most importantly, k a b 
do not depend on the fermion localizations alone. Whether or not there is a cancellation 
between the gauge and fermion KK contributions depends very much on the combination of 
the particular quark mass and mixing matrices considered just as well as the configuration 
of fermion localizations used. Such cancellation is by no mean generic, and has to be checked 
whenever a new combination of admissible configuration of fermion localizations, and quark 
mass and mixing matrices arise. In addition, since 5D gauge and Yukawa couplings are 
independent parameters, whether or not <C 1(^)^1 does not mean the same has 

to hold between |(«f b )i| and \(k{ b )n\- Since and k{- have very different structures (see 
Eqs. (1671) and (|68|) ). the combined effect when convolved with the particular quark mixing 
matrices can be quite different, as is the case for the three asymmetrical configurations listed 
in Table [H 

It is expected that both the single top and the it pair production rates will be high at the 
LHC, with the latter about a factor of two higher still than the former. To a small correction, 
the single tops are always produced in the LH helicity, while both helicities are produced 
in pair productions. Thus a simple way of testing the above at the LHC is to compare the 
decay rates of t — > Z + jets in single top production events (e.g. in the associated t W 
productions) to that from the pair productions, so that informations of both LH and RH 
decays can be extracted. Note that both the single and pair production channels should give 
comparable branching ratios initially at the discovery stage. Of course, a higher branching 
ratio would be obtained from pair productions after several years of measurements. 
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V. SUMMARY 



We have performed a detailed study of the admissible forms of quark mass matrices in 
the MCRS model which reproduce the experimentally well-determined quark mass hierarchy 
and CKM mixing matrix, assuming a perturbative and hierarchyless Yukawa structure that 
is not fine-tuned. 

We arrived at the admissible forms in two different ways. In one we examined several 
quark mass matrix ansatz which are constructed to fit the quark masses and the CKM 
matrix. These ansatz have a high degree of symmetries built in which allows the localization 
of the quarks (that give rise to the mass hierarchy in the RS setting) to be analytically 
determined. We found that the Koide-type symmetrical ansatz is compatible with the 
assumption of a hierarchyless Yukawa structure in the MCRS model, but not the Fritzsch- 
type hermitian ansatz. Because the ansatzed mass matrices are symmetrical, both LH and 
RH quark mixing matrices are the same. 

In the other way, no a priori quark mass structures were assumed. A numerical multi- 
parameter search for configurations of quark localization parameters and Yukawa couplings 
that give admissible quark mass matrices was performed. Admissible configurations were 
found after an extensive search. No discernible symmetries or pattern were found in the 
quark mass matrices for both the up-type and down-type quarks. The LH and RH mixing 
matrices are found to be different as is expected given the asymmetrical form of the mass 
matrices. 

We studied the possibility of differentiating between the case of symmetrical and asym- 
metrical quark mass matrices from flavour changing top decays, t — > Z + jets. We found 
the dominant mode of decay is that with a final state charm jet. The total branching ratio 
is calculated to be ~ 3 to 5 x 1CT 6 in the symmetrical case and ~ 9 x 10~ 6 to 2 x 1CT 5 in 
the asymmetrical case. The signal is within reach of the LHC which has been estimated to 
be 6.5 x 1CT 5 for a 5a signal at lOOfb -1 25|]. However, the difference between the two cases 
may be difficult to discern. 

We have also investigated the decay £r ; — > b^W as a large number of top quarks are 
expected to be produced at the LHC. We found a branching ratio at the level of C(10~ 5 ) is 
possible. Although the signal is not negligible, given the huge SM background, its detection 
is still a very challenging task, and a careful feasibility study is needed. This is beyond the 
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scope of the present paper. 
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Note added: After the completion of this work, we became aware of Ref. 28| which finds 



that flavour bounds from the AF = 2 processes in the meson sector, in particular that from 
6k, might require the KK mass scale to be generically O(10) TeV in the MCRS model. 



We will show in an ensuing publication 29( that parameter space generically exists where 
KK mass scale of a few TeV is still consistent with all the flavour constraints from meson 
mixings, and that our conclusions with regard to the top decay in this work continue to 
hold. 



APPENDIX A: THE HERMITIAN MASS MATRIX ANSATZ 

In this appendix we show that generically, the Fritzsch-type ansatz cannot be accommo- 
dated in the RS framework without requiring a hierarchy in the 5D Yukawa couplings. We 
consider below a general Hermitian mass matrix ansatz for which the Fritzsch-type ansatz 
is a special case of. 



1. General analytical structure 

The Hermitian mass matrix ansatz takes the form 

M f = PjMfPf , f = u,d, 
where Pf = diag{l, e^ c f : e l< ^<t>B f +<t>c f )^ j g a diagonal pure phase matrix, and 



> Vf=\V f \e iu ', u f = cf) Bf + (f> Cf - <j>v f , 



(Al) 







\c f \ 


Vf) 


M f = 


\c f \ 




\B f \ 




[v; 


\Bf\ 


Af ) 



(A2) 
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with (fix = arg(X) and Af, Df, Uf, \X\, (fix € R. Note that the Fritzsch-type ansatz with 
four texture zeroes [l5j is recovered when Uf — Vf — (the six-zero texture case 14| has 
Df = also). For simplicity, we take ojf G {0, n} below so that Vf = ±|V/|. 6 We will 
ignore the fermion label below for convenience. 

The matrix M can be diagonalized via an orthogonal transformation 



^ 0^ 



T MO 



A 2 
A 3 



|Ai| < |A 2 | < |A 3 



(A3) 



The eigenvalues |A$|, i = 1, 2, 3, can be either positive or negative. To reproduce the 
observed mass spectrum, we set |Aj| = mj. From the observed quark mass hierarchy, it is 
expected in general that \A\ be the largest entries in M, and \A\ < | A3 1 . Without loss of 
generality, we take A and A3 to be positive. 

By applying the Cayley-Hamilton theorem, three independent relations between the six 
parameters of M to its three eigenvalues can be deduced: 



S 1 -A-D-U = 0, 

5 2 + \B\ 2 + |Cf + V 2 - AD - (A + D)U = , 

5 3 + A\C\ 2 + DV 2 + U\B\ 2 - ADU - 2\B\\C\V = . 



Si — Aj , 

i 

$2=/, Ai Aj 

i<j 



(A4) 



Choosing A, U, V to be the free parameters, Eq. (1A4I) can be solved for \B\, \C\, and D: 



D = Si — A — U . 



\B\ 



VY + Z 



y/{A - U)X - 2V(VY + Z) 



\C\ 



(A - U)X - 2V(VY - Z) 
(A - U) 2 + AV 2 



(A5) 



where 



X = U 3 + (A + 2U)V 2 - (U 2 + V 2 )S 1 + US 2 - S 3 
Y = A 2 + V 2 + (A + U)(U - + S 2 , 
Z = ^V 2 Y 2 + (U - A)XY - X 2 . 



(A6) 



6 Such case has been considered in Ref . [26[ , and was shown to be consistent with the current experimental 
CKM data. 
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If \U\, \V\ <C I Ax | <C \A\ so that Eq. flAll) is a perturbation of the Fritzsch four-zero texture 
ansatz, \B\ and \C\ can be expanded as 



\B\ 



± 



.4 



l + fE Tev J2 + 0(4, e 2 y ; 



+ 



1-^- + ^ 

A A 2 



^ + € fTe v R(A) + 0(el,e 



\C\ 



A 



where 



'/ 



U 
A 



V 
A 



R 



'Y UA-\ 



(A7) 



(A8) 



Given that we have taken A < X 3 and A, X 3 > 0, it is required that S 3 < (or A1A2 < 0) 
for I B I and \C\ to be real. This is consistent with the expectation from the considerations 
of Ref. [3]. In the limit U, V — ► 0, the exact Fritzsch four- zero texture ansatz is recovered. 

With M determined by the three free parameters which we chose to be A, U, and V, so 
are its eigenvectors. For each eigenvalue Aj, its associated eigenvector takes the form 

T 



V; 



\C\(A - K) - \B\V , V 2 - (A - Xi)(U - K) , \B\(U - \) - \C\V) . (A9) 



The orthogonal matrix O is then given by 

/, , i\ 
0= vi v 2 v 3 



V 



Vi 



V; 



Vi 



, * — 1,2,3, 



(A10) 



/ 



and the quark mixing matrix by V m i x = 0^(P u P^)O a 



2. Matching to the RS mass matrix 

To reproduce the Hermitian mass matrix ansatz (1A1I) by the RS mass matrix (1251) . we 
match them and solve for the parameters determining the RS mass matrix. For the purpose 
of checking if hierarchy arises in the 5D Yukawa couplings from the matching, we may start 
matching in either the up or the down sector. For simplicity, the fermion species label is 
ignored below. 

There are a total of 24 parameters in M RS to be determined: six fermion wave function 
values, F L (cf) and F R (c R ), nine Yukawa magnitudes, p^, and nine Yukawa phases, 0y, where 
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i,j = 1, 2, 3. Matching results in nine conditions for both the magnitudes and the phases. 
Thus all the Yukawa phases are determined by 0b,cs while six magnitudes are left as free 
independent parameters. These we chose to be Fl(c 3 ) and F R (c§), which are constrained 
by EWPT, and p n , p 2 i, p^i, p%2- The determined parameters are then the five Yukawa 
magnitudes: 

kL V 2 p u kL V\B\ p 21 

Pl3 ~ F^Fr^vwU^' p23 ~ F L ^)F R (c*)v w \C\^ 

kL A 
P33 ~ F L (4)F R (c§)^> 
\C\Vp u p 32 DV P21P32 , A11 x 

Pl2 — Tjl TT , P22 — I DMrY | , l Aii j 

\B\U pu \B\\C\ P31 

the nine Yukawa phases: 

012 = 0C , 013 = 0B + 4>C , 

022 = , 023 = 0B , 



'11 


= 0, 




'21 


= -</ 




'31 


= -</ 


•>b- 



?32 = -<PB , <P33 



0, (A12) 



and the four fermion wave function values: 



v Pu V P21 

Fn(c?) = -ttt^— , F R (c«) = ——^7 . (A13) 

vwF L (c%)p 31 v w F L (c%)p 32 

Note that there are only three independent Yukawa phases because the mass matrix ansatz 

is Hermitian. Note also that since fermion wave functions are always positive, V and thus 

U have to be positive implying that uj = 0. 

From Eq. flAllj) . in order for the Yukawa couplings to be of the same order, it is required 

that pu ~ P21 ~ P31 ~ P32 ~ P33, and 

\C\V DV 1 V 2 1 V A \B\ 

\B\U ' |B||C| ' fL4 U V \C\ ' V ; 

For generic sets of parameters we find |£> M |/|C U | ~ C(10 3 ) and | -Brf | / 1 Crf | ~ O(50). However, 
parameter sets that reproduce all entries of the CKM matrix and also the Jarlskog invariant 
to within two standard error can only be found if V u ~ U u and Va ~ lOC/^. Thus hierarchy 
in the 5D Yukawa couplings cannot be avoided if the Hermitian mass matrix ansatz (lAlj) is 
to be accommodated in the RS framework. 
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APPENDIX B: RS MATCHING OF THE SYMMETRIC ANSATZ 



In this appendix, we give analytical expressions for the parameters determined from 
matching the RS mass matrix (|25|) to the mass matrix ansatz (|2T)|) . Starting with the up 
sector, the determined parameters are the five up-type Yukawa magnitudes: 

„ = kL CI p u u = kL B u p| 

Pl3 F L (c Q3 )F R (c U3 ) v w i u ' P23 F L (c Q3 )F R (c U3 ) v w p& ' 

„ = kL A u 

P ' 33 F L (c Q3 )F R (c U3 )v w ' 

ju C u P11P32 u Ai P21P32 mi\ 

P12 = ~fT~t — » P22 = o — ' 

-C-'uC.u P31 J->u P31 

the nine up-type Yukawa phases: 



u 

'11 = 


-2^, 




012 = 


-51- 


°2 J 


013 = 


-51- 


°3 


u 

'21 = 


-5\- 


o 2 , 


022 = 


-2^2 , 




023 = 


-5 U 2 - 




u 

'31 = 


-5$- 


°3 , 


032 = 


-02 - 


™ 
<>3 > 


033 = 


-251 , 





(B2) 

and the four fermion wave function values: 

Fl(c Qi ) = F L (c Q3 )^ , F L (c Q2 ) = F L {c Q3 ) P -f , 

U M Pll P21 

^fl(cui) = T-i , L ^ — 4" , F R (c U2 ) = L r — 4~ • (B3) 

(CQ 3 ) Vjy F L (Cq 3 ) ^ p^ 2 

Next the down sector. Given the information on the up sector, the determined parameters 
are the six down-type Yukawa magnitudes: 

d _ Fl(cq 3 ) £d d _ C u £ d p^ d 



Pl1 F L (c Ql ) C d Psi ^uCdP^ 31 ' 



d F,.(<- Q( ) (',, ,i _ C u CdPll n d <S 1 L^O:;l ■-',/ ,/ ■-',/ l'- 2i ,1 

P12 — ~EHZ \ id P32 f 1 d I „u P32 ) 



? L {c Ql )\B d \ r *< U \B d \p^ 



,1 F L (CQ 3 ) C d d _Cy L ^dPll n d d ' -I" <>.; > I LJ <!\ ,/ \ J - J ,i\ I'll ,/ 1 , , 

p13 ~ F L ( CQl ) a/ 33 ~ e u A d pf/ 33> ^ « '' : « « - (bl) 

the nine down-type Yukawa phases: 

0n = -2^, 0f 2 = - 

021 = - <*2 ) 022 = -2#j , 023 = T ~ <*2 ~ <*3 ' 





Fl(c Q3 ) d p\\ 






P21 = 


Fl(cq 2 ) P32 pt x 


P31, 






F l (cq 3 ) A d d 


A d 


P21 


P22 = 


F L (c Q2 ) \B d \ p32 ~ 


W 


P31 




f l(cq 3 ) \B d \ d 


\B*\ 


P21 


P23 = 


F L (c Q2 ) A d P33 ~ 


A d 


Pn 


Xd 

- 5 2 , 


0? 3 = - 


Xd 

-5 X - 


Xd 
°3 , 



'31 



*? - St , 0^ 2 = vr - 5 d 2 - 5t , </4 = -251 , (B5) 
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and the three fermion wave function values: 

C d kL 1 _ \B d \ kL 1 A d kL 1 

MCDJ t^Fx^,)^' ^ MCDJ ^F^cgjp^' ^ lCDj v w F L (c Q3 )pi 3 - 

(B6) 

Note that there are only six independent up-type Yukawa phases and six for the down- 
type Yukawa phases since the mass matrix ansatz is symmetric. With the texture phases 
(*>i2 3 determined by fitting the CKM data, there are three more relations, i.e. Sf = 5" = 
and #2,3 = #23 — ^235 which further reduce the number of independent Yukawa phases from 
a total of 12 down to nine. 

In order to be consistent with EWPT (Sg ZbL i L /g ZbL i L ~ 0-01 7 ) an d to avoid too large a 
correction to the Peskin-Takeuchi S and T parameters, it is required that 0.25 < cq 3 < 0.4, 
qy 3 < 0.2, so that rrigauge J$ 4 TeV [11]. To have the theory weakly coupled for at least the 



first two KK modes, |As| < 4 is required also [20J. It follows that 2.70 < Fl(cq 3 ) < 4.27, 
F R (c Ua ) < 7.15, and p\f < 4, which when combined with Eqs. fflfll) . flBij) . (157)1 and (I35]l 
imply 



4.06 < F L ( C Q3)F /? (c l 7 3 ) < 30.57 , 0.53 < p u 33 < 4 , (B7) 



and 



4 1 < 0.UF L (c Q3 )F R (c U3 ) , 4 1 < 0.25F i (c Q 3)F i? (c c , 3 ) , (B8) 
P31 P31 

It It It 14 

^32 < 2-19 , P f P i, < 1.52 , ^ < 1.92 , P f P % < 1.99 , 
P31 P31 P31 P31 

It It It It 

^P3 2 <3.97, ^<4, ^ 2 < 3.86, ^3 < 4.15. (B9) 

H31 V31 P31 P31 

Observe from Eq. (1B3|) that the second generation SU{2)l doublet, Q 2 , is local- 
ized towards the UV (IR) brane if p\\j p\ is less (greater) than Fl(0.5 + e)/Ft,(cQ 3 ) 
(F L (0.5 - e)/F L (c Q3 )). Note that F L (0.5 ± e) « 1 =f e£x c 7r for e < l/(2Ar c 7r). We plot 
in Fig. [6] the critical value of P31/P21 below (above) which Q 2 is localized towards UV (IR) 
brane. The same logic shows that the first generation SU(2)l doublet, Qi, is generically 
localized towards the UV brane because of the suppression factor ^ U /C u ~ 10~ 2 (even if 

P31/P11 > 1)- 



7 The bound we adopted here is that from the PDG. Studies of similar model but differing details where a 



complete electroweak analysis was carried out have produced a more stringent bound, e.g. < 0.0025 27 1 
Such complete EWPT analysis, however, is beyond the scope of the present work. 
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FIG. 6: The critical value of p\l p\\ as a function of cq 3 in the range allowed by EWPT. For values 
of P31/P21 in the "UV" ("IR") region, cq 2 is greater (less) than 0.5. 
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